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REAL-TIME ACQUISITION OF CO-REGISTERED X-RAY AND OPTICAL IMAGES 

CROSS-REFERENCE TO RELATED APPLICATION 

This application claims the benefit of U.S. Provisional Application Serial No. 
60/460,086 (Attorney Docket No. 2003P05012US), filed April 3, 2003 and entitled 
"System and Method for Real-Time Acquisition of Co-Registered X-Ray and Optical 
(Fluorescence, Coherent, Diffused or Transmission) Images", which is incorporated 
herein by reference in its entirety. 

BACKGROUND 

The present disclosure is directed towards systems and methods for acquiring 
co-registered X-ray and optical imagery. Medical image data, for example, is typically 
desired and/or obtained through various types of imaging modalities. 

A co-pending U.S. Patent Application (Atty. Docket Nos. 2000P09022US01 & 
8706-538), entitled "System and Method for Highlighting a Scene Under Vision 
Guidance", which is subject to the same duty of assignment as the present application, 
discloses a system and method for processing coordinates of a target point in a 
captured image of a real scene, and converting the image coordinates into the 
coordinates of a light projector to illuminate the target point. In addition, issued U.S. 
Patent Nos. 6,227,704; 6,229,873; 6,447,163 and 6,473,489; which are also subject to 
the same duty of assignment as the present application, disclose systems and methods 
for acquisition of video and X-ray images. 



What is needed is a new system and method for real-time acquisition of co- 
registered optical (e.g., fluorescence, coherent, diffused or transmission) and X-ray 
images. Such a system could provide physicians with new imaging abilities for 
applications including, for example, Arthritis treatment monitoring. For such an 
application, an optical image could show the enzyme activities, while a co-registered X- 
ray image could allow a physician to exactly locate the activity in relation to bone 
structure. In this way, a physician could take advantage of both imaging systems at the 
same time, by viewing the superimposed images, and thereby be relieved from having 
to relate the two types of images in his/her mind. The desired system could allow the 
physician to visualize each of the optical or X-ray images alone or in combination with a 
co-registered image. The present disclosure addresses these and other related issues. 

SUMMARY 

These and other drawbacks and disadvantages of the prior art are addressed by 
an apparatus, method and program storage device for co-registration of multi-modal 
images in a three-dimensional environment. 

A first embodiment apparatus for co-registration of multi-modal images in a three- 
dimensional environment includes a source of excitation light, a one-way mirror having 
a transmissive side disposed towards the excitation light for transmitting the excitation 
light and a reflective side for reflecting light received from a target, an electromagnetic- 
ray source disposed relative to the source of excitation light, an electromagnetic-ray 
transparent mirror having a light-reflective surface disposed towards a reflecting side of 
the one-way mirror and an electromagnetic-ray transmissive surface disposed towards 
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the electromagnetic-ray source, a target location disposed towards the light-reflective 
surface of the electromagnetic-ray transparent mirror for locating a target and receiving 
the excitation light and the electromagnetic-rays, an electromagnetic-ray detector 
disposed on an opposite side of the target location relative to the electromagnetic-ray 
source for detecting electromagnetic-rays transmitted through the target, and a light 
detector disposed towards the reflective side of the one-way mirror for detecting light 
from the target. 

A second embodiment apparatus for co-registration of multi-modal images in a 
three-dimensional environment includes a source of excitation light, an electromagnetic- 
ray source disposed relative to the source of excitation light, an electromagnetic-ray 
transparent mirror having a first surface disposed towards the excitation light and a 
second surface disposed towards the electromagnetic-ray source, a target location 
disposed towards the first surface of the electromagnetic-ray transparent mirror for 
locating a target and receiving the excitation light and the electromagnetic rays, an 
electromagnetic-ray detector disposed on an opposite side of the target location relative 
to the electromagnetic-ray transparent mirror for detecting electromagnetic-rays 
transmitted through the target, a second electromagnetic-ray transparent mirror having 
a light-reflective surface disposed towards the target location, and a light detector 
disposed towards the light-reflective surface of the second electromagnetic-ray 
transparent mirror for detecting light from the target. 

These and other aspects, features and advantages of the present disclosure will 

C 

become apparent from the following description of exemplary embodiments, which is to 
be read in connection with the accompanying drawings. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present disclosure teaches an apparatus and method for image 
segmentation in a three-dimensional environment, in accordance with the following 
exemplary figures, in which: 

Figure 1 shows a schematic diagram of an optical imaging system where the 
source of the excitation light and the optical camera detecting the emitted or reflected 
light have the same projection geometry; 

Figure 2 shows a schematic diagram of an optical imaging system where the 
source of the excitation light and the optical camera detecting the emitted or reflected 
light have the same projection geometry, and are each on the same side and attached 
side-by-side; 

Figure 3 shows a schematic diagram of a co-registered imaging system in 
accordance with an illustrative embodiment of the present disclosure where the 
excitation is done with a source of coherent light; 

Figure 4 shows a schematic diagram of a co-registered imaging system in 
accordance with an illustrative embodiment of the present disclosure where the detector 
of the transmitted light is positioned such that its images are co-registered with X-ray 
images; 

Figure 5 shows a schematic diagram of a co-registered imaging system in 
accordance with an illustrative embodiment of the present disclosure where the detector 
of the transmitted light is positioned such that its images are co-registered with X-ray 
images and the detector is attached to the side of the X-ray detector; and 
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Figure 6 shows a schematic diagram of a co-registered imaging system in 
accordance with an illustrative embodiment of the present disclosure where the system 
is capable of rotating by at least 200 degrees around a patient. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

The present disclosure introduces a system and method for real-time acquisition 
of co-registered X-ray and optical images, including, for example, fluorescence, 
coherent, diffused or transmission images. System embodiments provide physicians 
with new imaging capabilities applicable to many applications, including, for example, 
Arthritis treatment monitoring. 

In an Arthritis treatment monitoring example, the optical image shows the 
enzyme activities. The co-registered X-ray image allows the physician to precisely 
locate the enzyme activities in relation to the bone structure of the patient. In this way, 
the different types of images are superimposed so that the physician may take 
advantage of both imaging systems at the same time without the need to relate the two 
images manually. A preferred system embodiment permits the physician to visualize 
each of the optical or X-ray images alone, or combined into one co-registered image. 

In the description that follows, an X-ray or X-ray fluoroscopy system is illustrated 
with its two main components: a) X-ray source; and b) X-ray detector. An optical 
imaging device is also illustrated by its two main components: a) source of excitation 
light; and b) detector of emitted, reflected, diffused or transmitted light. 

As shown in Figure 1 , a first exemplary optical imaging system is indicated 
generally by the reference numeral 100. The imaging system 100 includes a source of 
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excitation light 110 projecting through a non-reflecting side of a one-way mirror 120 to a 
target 130. The light emitted or reflected from the target 130 is reflected by a reflecting 
side of the one-way mirror to a detector of emitted or reflected light 140. Thus, in the 
optical imaging system 100, the source of the excitation light and the optical camera 
detecting the emitted or reflected light have the same projection geometry. 

Turning to Figure 2, a second exemplary optical imaging system is indicated 
generally by the reference numeral 200. The imaging system 200 includes a source of 
excitation light 210 projecting through a non-reflecting side of a one-way mirror 220 to a 
target 230. The light emitted or reflected from the target 230 is reflected by a reflecting 
side of the one-way mirror to a mirror 222, which reflects the light to a detector of 
emitted or reflected light 240. Thus, in the optical imaging system 200, the source of 
the excitation light and the optical camera detecting the emitted or reflected light have 
the same projection geometry. In addition, the source and the detector are on the same 
side and can be attached side-by-side. 

Turning now to Figure 3, a first exemplary co-registered imaging system is 
indicated generally by the reference numeral 300. The imaging system 300 includes a 
source of excitation light 310 projecting through a non-reflecting side of a one-way 
mirror 320 to a mirror 360. The mirror 360 is an X-ray transparent mirror that reflects 
the light to a target 330. The light emitted or reflected from the target 330 is reflected by 
the mirror 360 to a reflecting side of the one-way mirror 320, which reflects the light to a 
detector of emitted or reflected light 340. The imaging system 300 further includes an 
X-ray source 350 projecting through the X-ray transparent mirror 360 to the target 330. 
An X-ray detector plate 370 receives the X-rays transmitted through the target 330. 
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Thus, the resulting system 300 provides co-registered X-ray and emitted or 
reflected images. In a case where the excitation is done with a source of coherent light, 
the detector detects the reflected light. In a case of fluorescence imaging, the detector 
will detect the emitted light from the object. Note that different detectors and special 
filters may be used for each of the imaging systems described herein. 

As shown in Figure 4, a second exemplary co-registered imaging system is 
indicated generally by the reference numeral 400. The imaging system 400 includes a 
source of excitation light 410 reflecting from a mirror 422. The light reflected from the 
mirror 422 to a mirror 460. The mirror 460 is an X-ray transparent mirror that reflects 
the light to a target 430. The light transmitted through the target 430 is reflected by 
another X-ray transparent mirror 462, which reflects the light to a detector of transmitted 
light 442. The imaging system 400 further includes an X-ray source 450 projecting 
through the X-ray transparent mirror 460 to the target 430. The light transmitted 
through the target 430 is further transmitted through the other X-ray transparent mirror 
462. An X-ray detector plate 470 receives the X-rays transmitted through the target 430 
and the other X-ray transparent mirror 462. Thus, in the system 400, the detector of the 
transmitted light is positioned in a way that the images it provides are co-registered with 
X-ray images. 

Turning to Figure 5, a third exemplary co-registered imaging system is indicated 
generally by the reference numeral 500. The imaging system 500 includes a source of 
excitation light 510 reflecting from a mirror 522. The light reflected from the mirror 522 
to a mirror 560. The mirror 560 is an X-ray transparent mirror that reflects the light to a 
target 530. The light transmitted through the target 530 is reflected by another X-ray 
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transparent mirror 562, which reflects the light to a detector of transmitted light 542. 
The imaging system 500 further includes an X-ray source 550 projecting through the X- 
ray transparent mirror 560 to the target 530. The light transmitted through the target 
530 is further transmitted through the other X-ray transparent mirror 562 to another 
mirror 524. An X-ray detector plate 570 receives the X-rays transmitted through the 
target 530, the other X-ray transparent mirror 562 and the other mirror 524. Thus, in the 
system 500, the detector of the transmitted light is positioned in a way that the images it 
provides are co-registered with X-ray images, and this detector can be conveniently 
attached to the side of the X-ray detector. 

Turning now to Figure 6, a fourth exemplary co-registered imaging system is 
indicated generally by the reference numeral 600. The imaging system 600 includes a 
source of excitation light 610 reflecting from a mirror 622. The light reflected from the 
mirror 622 to a mirror 660. The mirror 660 is an X-ray transparent mirror that reflects 
the light to a target 630. The light transmitted through the target 630 is reflected by 
another X-ray transparent mirror 662, which reflects the light to a detector of transmitted 
light 642. The imaging system 600 further includes an X-ray source 650 projecting 
through the X-ray transparent mirror 660 to the target 630. The light transmitted 
through the target 630 is further transmitted through the other X-ray transparent mirror 
662 to another mirror 624. An X-ray detector plate 670 receives the X-rays transmitted 
through the target 630, the other X-ray transparent mirror 662 and the other mirror 624. 
The entire system 600, excluding the target 530, is mounted to gimbals, tracks or like 
devices for rotating about the centrally disposed target 530. 
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Thus, in a particularly preferred embodiment, the co-registered X-ray and 
Transmitted Optical Imaging system 600 rotates around the target or patient by at least 
200 degrees, while acquiring X-ray and transmitted optical images. These images are 
then used for tomographic reconstruction. The two sets of resulting reconstructed 3D 
data are thus fully co-registered by design. This allows the physician to visualize the 
data as two separate data sets or one composite data set. 

According to one aspect of the present disclosure, the image capture device and 
the illumination device comprise common optical properties. This may be realized by 
means of one one-way mirror. Figure 1 provides an exemplary construction of such a 
system. Figure 2 provides a second exemplary construction where the source and the 
detector can be attached side-by-side using an additional mirror. 

In another aspect of this present disclosure, the system allows the acquisition of 
real-time X-ray and fluorescence or coherent optical imaging. Figure 3, illustrates an 
exemplary construction of such a system. The system allows the user to choose a 
target to be illuminated in a real scene using X-ray image data of the same scene. 

In one aspect, a method for illuminating a target point in a real scene comprises 
the steps of capturing X-ray image data of a scene, identifying X-ray image data 
associated with a target point, and projecting a light beam at the target point in the rear 
scene. The step of projecting comprises the steps of converting image coordinates of 
the target point to light coordinates for directing the light beam, and processing the light 
coordinates to direct the light beam to the target point in the real scene. 

The illumination system and its control for embodiments of the present disclosure 
may be realized as described in the co-pending U.S. Patent Application (Atty. Docket 
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Nos. 2000P09022US01 & 8706-538), entitled "System and Method for Highlighting a 
Scene Under Vision Guidance". A difference here is that the system can also be guided 
by X-ray vision in cases where the target for illumination can be better defined in 
respect to the anatomical targets visible under X-rays. Once the target is illuminated, 
the reflected or the emitted light, in the case of fluorescence imaging, is captured by an 
optical sensor with the same imaging geometry as the X-ray imaging system. This 
allows the user, such as a physician, to see a composite image, which includes both 
information from X-ray and optical imaging. Such system embodiments can provide 
new imaging possibilities, which can play an important role in improving the diagnosis 
and treatment monitoring procedures for many diseases. 

Figures 4 and 5 illustrate two other aspects of the present disclosure. In these 
cases the system co-registers the X-ray and transmitted optical imaging. The 
Transmitted Optical Imaging here is defined as a system composed of a light source, 
which emits the light rays towards an object of interest and a detector on the other side 
of the target, which measures the intensity of the transmitted light. This intensity is a 
function of the absorption of the tissues and/or material forming the object (for example 
see "Imaging through Random Media Using Low Coherence Optical Heterodyning" by 
A. Schmidt, R. Cotey, and P. Saulnier, Optics Letters, Volume 20, Number 4, February 
15, 1995). System embodiments of the present disclosure can use two, three or four 
mirrors, for example, depending on the choice of having the light source and transmitted 
light optical detector on the side, or orthogonal to, the X-ray source and detector, 
respectively. 
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According to another aspect of this present disclosure, the co-registered X-ray 
and Transmitted Optical Imaging system rotates around the patient or animal target by 
at least 200 degrees, while acquiring both X-ray and optical images. These images can 
then be used for tomographic reconstruction. The two sets of resulting reconstructed 
3D data are thus fully co-registered. This allows the physician to visualize the data as 
two separate or one composite data set. The calibration and reconstruction from 
rotating C-arms have been extensively discussed in the literature. For example, the 
following U.S. Patents propose different methods for calibration and tomographic 
reconstruction using such system: US-6,049,582; US-6,038,282; US-5,963,613; US- 
5,963,612; US-5,923,727; US-5,835,563 and US-5,822,396. In preferred embodiments 
of the present disclosure, a particularly advantageous feature is in the co-registration of 
the X-ray and Transmitted Optical Imaging systems by the particular construction and 
calibration of this system. This allows obtaining co-registered 3D reconstruction data 
from these two modalities. 

The relative placement of the elements of embodiments of the present disclosure 
are important. The X-ray imaging system, the optical imaging system, and the mirrors 
need to be placed correctly, as will be understood by those of ordinary skill in the 
pertinent art. There is also a feature for computing a transformation, taking into account 
the differences in the intrinsic imaging parameters of the X-ray and Optical images. 
These geometrical and imaging calibrations can be considered as modifications and 
improvements of those described in U.S. Patent Nos. 6,473,489; 6,447,163; 6,229,873 
and 6,227,704, for example. 
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These and other features and advantages of the present disclosure may be 
readily ascertained by one of ordinary skill in the pertinent art based on the teachings 
herein. It is to be understood that the teachings of the present disclosure may be 
implemented in various forms of hardware, software, firmware, special purpose 
processors, or combinations thereof. 

Most preferably, the teachings of the present disclosure are implemented as a 
combination of hardware and software. Moreover, the software is preferably 
implemented as an application program tangibly embodied on a program storage unit. 
The application program may be uploaded to, and executed by, a machine comprising 
any suitable architecture. Preferably, the machine is implemented on a computer 
platform having hardware such as one or more central processing units ("CPU"), a 
random access memory ("RAM"), and input/output ("I/O") interfaces. The computer 
platform may also include an operating system and microinstruction code. The various 
processes and functions described herein may be either part of the microinstruction 
code or part of the application program, or any combination thereof, which may be 
executed by a CPU. In addition, various other peripheral units may be connected to the 
computer platform such as an additional data storage unit and a printing unit. 

It is to be further understood that, because some of the constituent system 
components and methods depicted in the accompanying drawings are preferably 
implemented in software, the actual connections between the system components or 
the process function blocks may differ depending upon the manner in which 
embodiments of the present disclosure are programmed. Given the teachings herein, 
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one of ordinary skill in the pertinent art will be able to contemplate these and similar 
implementations or configurations of the present disclosure. 

Although the illustrative embodiments have been described herein with reference 
to the accompanying drawings, it is to be understood that the present disclosure is not 
limited to those precise embodiments, and that various changes and modifications may 
be effected therein by one of ordinary skill in the pertinent art without departing from the 
scope or spirit of the present disclosure. All such changes and modifications are 
intended to be included within the scope of the invention as set forth in the appended 
claims. 
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